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ABSTRACT 

The PRINCE materials for undergraduate collegiate 
students, developed at Syracuse University, were field tested at six 
higher education institutions for this evaluation study* The 
materials, consisting of four versions of a simulation, vere designed 
to teach skills for analyzing political situations from a strategic 
point of view, vith the objective of improving political skills. This 
study evaluated the materials for teaching specific skills to 151 
students by measuring posttest improvements over pretests. Open-ended 
essay questions vere used as the evaluation instrument, and ansveri^ 
to the questions by a control group of politically experienced 
persons vere used for determining criteria. The results indicated 
tha^ students tended to (1) be more likely to define a clear-cut 
political issue, (2) vrite fever vords in defining the issue, and (3) 
make more specific suggestions as to hov to deal vith more actors in 
trying to solve a political problem. The findings generally shoved 
that the stipulated educational objectives vere being achieved at the 
campuses tested. (HD) 
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INTRODUCTION AND OVERVIEW* 



This paper presents the findings of a project supported by 
'the EXXON Foundation for Higher Education to evaluate a set of 
educational materials whose primary purpose was to develop the 
ability of undergraduates to analyze a political situation from 
a strategic point of view. The evaluated materials consist of 
four versions of a simulation: a paper and pencil version in a 
book, a board game, a roan-computer exercise, and an all computer 
model. Each item can be used either independently or in conjunc 
tion with any of the others. The materials, developed at Syracu 
University, were tested at six other institutions by faculty mem 
bers who agreed to participate in the evaluation study. 

The information upon which the findings were based were ob- 
tained from an extended response or open-ended essay examination 
given on a pre-test/post-test basis at the six participating in- 
stitutions. A total of 151 undergraduate students at the six 
campuses used at least one form of the materials and took the 
pre and post tests. A criterion standard against which the stu- 
dent responses could be judged was provided by the responses to 
the same test from a group of respondents assumed to possess mod 
erate to high political skills because of their vocations. 

Analysis of the data yielded findings indicating an improve 
ment in students* political skills, as measured by four indica- 
tors (although the improvement in one of the four was not statis 
tically significant at the .05 level). In all cases. there was a 
change in post- test responses by the students such that the an- 
swers were more similar to the criterion respondents than were 
the pre-test responses. Gain was said to. have occurred if, on 
the post-test, the student scored higher in (1) defining polit- 
ical issues described in the scenario as the stimulus for the 
essay; (2) identifying the political factors that would deter- 
mine the outcome of a given issue more clearly; and (3) suggest- 
ing more complex strategies that might be applied in resolving 
the issue. The remainder of this paper is organized into three 
sections: (1) a description of the materials; (2) an explana- 
tion of the methodology and sampling procedures; and (3) a pre- 
sentation of the findings. 



We wish to thank Lawrence Taulbee, David Ahola, Alfred 
Arkley, Grant Hammond, George Balch, Jonathon Wilkenfeld, Joseph 
Spina, Thomas O'Donnell, John Handel man, and Robert Donaldson 
for conducting the evaluation and offering helpful comments dur- 
ing the preliminary stages of this paper,. 
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DESCRIPTION OP THE MATERIALS USED IN 
TEACHING THE SKILLS 



EMC 



The educational materials which were evaluated were devel- 
oped from the PRINCE Project. These materials constitute a mul- 
tidimensional introduction to politics, with an emphasis on en- 
couraging students to put their knowledge to work in order to ac- 
complish political goals. The materials stress a systematic ap- 
proach to the study of politics. At the same time, they are eas- 
ily used by even the most inexperienced students, so they can move 
to their own desired level of skilled political analysis. The 
content of the materials is flexible enough to allow for a focus 
on politics at the international, cross-national, national, or 
local levels. The materials can be used with differing levels 
of methodological sophistication — ranging from logical qualita- 
tive analysis through systematically structured manual calcula- 
tion, including statistical and computer simulation methods. 
The project under which the materials were developed has been sup- 
ported by the Office of Education, Department of State, National 
Aeronautics and Space Administration, the National Science Founda- 
tion, and the Exxon Foundation of Higher Education. The latter 
supported the specific research reported in this paper. 

The first educational package is a conputer simulation model 
called PRINCE, a programmed international conputer environment. 
PRINCE serves as an educational tool which students use in build- 
ing a structured image of contenporary international relations. 
Students use the conputer simulation as a man-machine exercise 
in which they assume the role of United States policy-makers 
while the computer simulates five other nations and eleven do- 
mestic pressures operating on American foreign policy-makers. 
Students are introduced to the exercise through a participant's 
guide. After the initial play, the students are introduced to 
the conceptual structure of the model and required to relate ma- 
jor theoretical and empirical studies in the field to that struc- 
ture.^ The major educational aim of PRINCE is to develop decision- 

^The entire set of research and educational activities asso- 
ciated with the "Educational Research arid Pol icy-Making Activi- 
ties of the PRINCE Project," Policy Studies Journal . Vol. 2, Sum- 
mer, 1974, William D. Coplin and Michael K. O'Leary (Urbana, 111.: 
University of Illinois Department of Political Science and Insti- 
tute of Government and Public Affairs) . 

2 

William D. Coplin, Michael K. O'Leary, Stephen L. Mills, 
PRINCE Participant's Guide; Concepts, Environments, Procedures , 

(New York: Learning Resources in International Studies, 1973). 
3 

William D. Coplin, Michael K. O'Leary, Stephen L. Mills, 
John R. Handleman, "A Description of the PRINCE Model," (New York: 
Learning Resources in InternaJbipnal Studies, 1973) . 




making skills by having interact with a complex foreign 

policy environment created^Plfhe computer. 

A second educational component of the project and the one 
most widely used is a book entitled Everyman's PRINCE; — A Guide 
to understanding Your Political Problems .^? The. book presents a 
simple model of the political process using many of the variables 
contained in PRINCE. It presents that model in the form of a 
"political accounting system" or tool through which any individual 
can analyze political situations and generate strategies. Stu- 
dents are introduced to the model as a problem-solving tool and 
then provided with a series of case studies. The case studies 
describe historical (e.g., the Constitutional Convention) and 
hypothetical situations using the basic conceptions and assump- 
tions of the model. Each of the case studies is presented as a 
simple simulation run of the model. Students are encouraged to 
follow-up their reading of the book with paper and pencil sim- - 
ulation exercises, applying the application model to a political 
problem of their choice. 

In addition to the PRINCE man-computer international rela- 
tions simulation and to the book Everyman's PRINCE, a computer 
analysis program, called PROBE, 5 (Policy Research, Observation 
and Evaluation) has been developed. The purpose of PROBE is to 
take information generated by the application of the PRINCE Po- 
litical Accounting System and predict policy decisions. PROBE 
is based on a reference group theory of politics. & This theory 
involves the interaction of the output actor with the reference 
actors. The interaction of the reference actors is crucial in 
determining the decisions of the output actor and hence, the 
resolution of the problem. 

The fourth educational medium of the PRINCE Project is a 



William D. Coplin, Michael K. O'Leary, Everyman's PRINCE; 
A Guide to Understanding Political Problems (North Scituate; 
Duxbury Press, 1973) . 

^Michael K. O'Leary, William D. Coplin, Gary Brey, Sharon 
Dyer, PROB E User's Ma nual (New York; Learning Resources in In- 
ternational Studies, 1974) . 

^William D. coplin, Michael K. O'Leary, "A Reference Group 
Theory of Politics" (Syracuse International Relations Program, 
1974) . 
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board^game called PRINCEDOWN. It can be applied to a wide va- 
riety of political problems. Students first do research to col- 
lect information on a given set of issues. This information then 
becomes the scenario for the game. Profiles are likewise pre-- 
pared for each actor, and students play their roles according to 
the profiles and the research they have already completed. 
PRINCEDOWN provides an action framework to explore the strategy 
implication of the political situations pictured in the account- 
ing system. 

Before presenting the evaluation of these materials, three 
general points about educational objectives should be made. 
First, while the four types of PRINCE materials were designed to 
improve the student's ability to think strategically about po- 
litical situations, each emphasizes different skills. The book, 
Everyman's PRINCE , is primarily designed to transmit the concep- 
tual framework. PRINCE, the man-machine simulation is intended - 
to improve the student's ability to formulate and implement pol-- 
icy in a complex environment. PROBE provides a framework for' 
forecasting. PRINCEDOWN develops negotx.ition and bargaining 
skills. The purpose of this evaluation, however, is to evaluate 
the general skill of thinking about politics in strategic terms. 
Our comments throughout will reflect this emphasis, and we will 
not be concerned with the secondary, although equally important, 
objectives of each of the four forms of the materials. 

The second point concerning our objectives is that we are 
attempting to provide a set of competencies that can be applied 
by students in a variety of political contexts ranging from those 
events and conditions that are described in most newspapers to 
circumstances that will confront them personally in everyday life 
To paraphrase the preface to Everyman's PRINCE , we believe that 
all citizens should be able to identify how their lives are af- 
fected by the collective decisions of formal or informal groups. 
Furthermore, they should know how to analyze the processes lead- 
ing to those decisions in order to be able to influence them. 
These general objectives have been found comprehensible and ap- 
pealing by both conventional liberal arts undergraduates and by 
a large variety of other students and professional trainees. 

^William D. Coplin, Michael K. O'Leary, Stephen L. Mills, 
^•PRINCEDOWN Student xManual : A Ganing Approach to the Study of 
Policy Issues (New York: Learning Resources in International 
Studies, 1972) . 
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Police cadets^ civil servants, foreign service officers, sales- 
men, business executives and community groups like the League of 
Women Voters and neighborhood associations have been enthusiastic 
about the materials. We believe that the materials represent di- 
verse social technologies for dealing with an important set of 
conditions faced by people in all walks of life. It is our be- 
lief that the skilled professional in almost every field can 
profit from development of the political skills which these ma- 
terials seek to develop. 

Part of the evidence for this belief stems from the occasional 
negative reaction of individuals in positions who possess the ba- 
sic political skills we seek to impart. On occasion, someone in 
a training session will criticize the materials for presenting 
"obvious" and "commonplace" ideas. These critical individuals, 
as it happens, are almost invariably the most experienced and 
skillful political practitioners in the group. Our response has 
been that, indeed, for a sophisticated political practitioner, 
the material is quite simple and elementary. However, for most 
people — especially liberal arts students and junior level pro- 
fessionals — the political skills we try to teach are far from 
obvious. This response of the politically aware to the materials 
as well as the results of the "criterion respondents" described 
on page 11 demonstrate, we believe, the general value of this ed- 
ucational objective to a broad range of individuals. 

The third general point about objectives is related to the 
fear that has sometimes been expressed about the educational ob- 
jectives of the PRINCE materials: it may be socially harmful to 
train students in the processes of "expedient" thought concern- 
ing political strategy in the absence of any concern for norma- 
tive consequence of proposed courses of action.® Quite frankly, 
however, it is our view that a good deal more value-mongering 
goes on in political science classes than does teaching in the 
processes of effective political strategy. In the larger social 
sense, the objectives we have are: (1) to train students as fu- 
ture citizens to be able to understand the processes used by 
leaders in making policy and (2) to train those few students who 



John Vasquez, "Toward a Unified Strategy for Peace Educa- 
tion: Resolving the Two Cultures Problem in the Classroom," 
Paper prepared for the 1975 Annual International Studies As- 
sociation Meeting, Washington, D.C,, Febaniary 18-22, 1975, 
pp. 5-6. 
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will become leaders in political and professional fields to make 
politically effective and uormatively inforraed policy. To a 
large extent^ we think that by helping students to think about 
ways of achieving their political goals, we are contributing to 
the general goal of many educators in value clarification rather 
than value acquisition. 



METHODOLOGY 



At the outset, we will attempt to state as clearly and pre- 
cisely as possible our criteria for measuring the students' acqui- 
sition of skills of political strategy. 

We developed these criteria on the basis of our own research 
in the policy-making process and, more specifically, through the 
administration of a questionnaire -on political strategy-to a-set-- 
of "criterion respondents," people who held elective or appointed 
political and administrative positions which, by their very na- 
ture, suggested the possession of political skills. ^ As a re- 
suit, we have identified three specific skills which taken to- . 
gether constitute the general political strategy capacity for \ ' 
which we seek to train students. 

The first of these skills is the ability, in a direct and 
concise fashion, to identify political issues in a given situa- - 
tion. Frequently, students (and others who have not had much 
political experience) tend to comprehend the features of a po- 
litical situation in an undifferentiated wholistic way, so that 
they are not able to distinguish among disagreements about fore- 
casts about the consequences of a given oolicy, the conflicting 
interests of actors, and their own value preferences. The abil- 
ity to isolate the political issue, we believe, is the essential 
first step in political analysis. 

The second skill in political strategy is the ability to 
determine the political factors that shape the bargaining over 
how the identified issue is to be resolved. Students shv>uld be 



^E.g., see the discussion by F. M. Newmann and D. W. Oliver, 
Clarifying Public Controversy (Boston: Little, Brown & Company, 
1970)/ Chapter 1. We are in.:^ebted to Professor Alfred Arkley 
for this comment and the citations in this footnote. 

""■^e will discuss this criterion-setting procedure in more 
detail below. 
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able to ascertain the set of actors whose interests are at stake 
and whose behavior will affect the outcome of the issue. This is 
in part a level of analysis skill since it requires students to 
separate background factors such as aggregate economic conditions 
from an analysis of involved individuals and groups. The back- 
ground factors should be analyzed as they shape the behavior of 
the group but should clearly be viewed as mediated by the actors. 

The third component of political strategy is the apprecia- 
tion of the use of indirect as well as direct action in order to 
achieve a goal. Simple admonitory solutions such as "Mr. X 
should make the most effective decision" do not adequately deal 
with the politics of the situation. Instead, students should 
carefully dalineate how each of the critical actors might react 
and how each can be influenced. Dealing as specifically as pos- 
sible with each of the major actors in any given situation wduld 
demonstrate the kind of complex political analysis that we ue- - 
lieve is necessary for the formulation and implementation ' of an-^--- 
effective political strategy. .We feel that this formulation of ^' 
political strategy skills represents one of the minimal capabil- 
ities that should be possessed by individuals in their vocational 
and citizenship roles. It represents a domain of competence pos- 
sessed by virtually everyone with substantial political experi- 
ence. Skillful participants in political and bureaucratic battles 
(whether in the public or private sector) must be able to perform 
these analyses. We recognize that this is a minimal level of so- 
phistication; more advanced political strategy skills could also" 
be identified. However, at this point we would be satisfied if 
we could demonstrate that there is a practical way to help stu- 
dents to define political issues, to identify political factors, 
and to develop relatively complex political strategies for deal- 
ing with essential political actors. 

The Research Instrument 

Our first major decision in developing our instrument was ~| 
to employ an essay rather than objective test. After some pre- 
liminary analysis, we concluded, along wich others, that while 
objective tests are "suitable for testing memory of factual de- 
tails, " they are less suitable for measuring the higher mental 
processes of logical reasoning, critical evaluation or creative 
synthesis which are the kinds of information we were seeking. 

"'■"'■R. L. Ebel, Measuring Educational Achievement . (Englewood 
Cliffs, N.J.: Prentice-Hc.il, Inc., 1965). Chapter 4, "The Char- 
acteristics and Uses of Essay Tests." 
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In contrast, we viewed essay tests as providing the student with 
the opportunity to "originate ideas, to organize and express 
ideas, and to integrate ideas in a global attack on the prob- 
lem. " It seemsd to us that the political skills we have ider*- 
tified require the kinds of activities which can be better mea- 
sured by an essay test. 

Figure 1 and Figure 2 contain the two versions of the tests 
that were employed in the project. Both are based on a news" 
story taken from the New York Times . They ask the student to 
identify the political issue and to provide guidance to the rel- 
evant political actor on what strategies he might undertake to 
gain his objectives. The cues provided in the question are pur- 
posely ambiguous. We endeavored to avoid any materials-bias in 
testing the student's ability to define the political issue, to 
identify the operating political factors, . and. td. suggest strate- 
gies that the actor in question might follow. The news story/. 
provides a stimulus and the questions provide a' generalized op- : - 
portunity for the student to react to the stimulus 

The coding scheme that appears in Appendix *A of this paper 
was developed for transforming the essay answers into quantita- 
tive variables, that would allow for statistical analysis. That 
scheme was applied by undergraduate^ who were trained with PRINCE 
material, but were not told the hypotheses being tested. Ques« 
tions of reliability and validity were explored throughout the 
data collection period, - • 

Reliability checks between two different coders provided 
inter-coder agreement scores ranging from 50 to 100 percent de- 
pending on the item in the scheme. Because the reliability was 
not high on a number of the more crucial and annbiguous items, a 
double coding of each test answer was undertaken and disagree- 
ments were settled by a third coder. To make sure that the cod- 
ers were not biasing their decisions by guessing whether the 
tests were pre or post, coders were asked to recode a random 
sample of 20 of their previously coded tests. That recoding re- 
sulted in an average inter-coder agreement of 90% or better across 
13 

Items." 



N. E. Gronlund, Measurement and Evaluation in Teaching . 
(New York: The Macmillan Company, l97lj. Chapter 10, "Measuring 
Complex Achievement: The Essay Test." 

""■■^Since each coder had coded at least 100 answers, we felt 
confident that they did not recognize the masked response they 
were asked to reevaluate. 
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Article, "Peris Again Balks a Europe Accord: Opposes Common Market Proposal 
for Consultation with U»S. on Policy," from the New York Times removed 
to conform with copyright laws* 



!♦ Assume that you are an advisor to an American government official. 
State for him the main political issue discussed in the article. 
Make your answer as specific as possible. 



2. What strategy would you suggest the American government pursue 
with raspect to this situation? Assume that you are providing 
recommendations for the American government. Hake your response 
as pointed as possible, and make sure you are clear in what actions 
you recommend . 
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SJJ.N.Y. Chief Asks Tenur e Analysis 



Article, "S.U.N.Y. Chief Asks Tenure Analysis: Walking a Tightrope" by Iver Peterson, 
from The New York Times removed conform with copyright laws. 



i.rter reading the article from T he New York Times , please answer the 
following two questions. 



1. Assume you are an advisor to Chancellor Boyer. State for him the 

main political issue discussed in the article. Make your answer as 
specific as possible. 



ERLC 



2. What strategy would you suggest that Chancellor Ernest L. Boyer 
pursue in handling the situation? Assume that you are providing 
recommendations for Chancellor Boyer. Make your response as pointed 
as possible, and make sure you are clear in what actions you recommend. 
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Validity questions are less tractible to statistical tests 
than reliability problems. Our principal effort to solve this 
problem was to obtain "criterion answers" to our two tests from 
respondents who had a substantial amount of political experience. 
Two groups of individuals were used. The first was a set of 
nineteen political figures primarily from the city of Syracuse 
and New York State. A few federal officials were also included 
in this sample. The second group consisted of twenty-one middle- 
level managers from the Department of Social and Health Services 
in the state of Washington, This second group was originally 
scheduled to be a test group. But when we were fully informed 
of their experiences, it became clear that they were more appro- 
priately considered to be a criterion group. The coding rules 
and the types of variables employed in the study were in large 
part developed through analysis of these criterion tests. We 
assumed that these 'individuals possessed a high degree of the 
political skills which we are trying to teach. We hypothesized 
that after an educational treatment, students' answers would more, 
closely approximate the ' criterion answers. But we also ^expected ' 
that, if this whole enterprise had validity, the highly experi- 
enced respondents would score better than the trained students. ~ 
This is because the criterion respondents, as a result of their 
political experiences would establish high standards which the 
trained students could approach, but rarely achieve, after only 
a relatively few hours of exposure to the materials. 

We will now briefly describe the coding rules "to determine/^- 
the relative performance of the students on each of the three 
skills. The reader should consult Appendix A for a full descrip- 
tion of the coding rules and forms used by the coder. 

In determining whether or not the student was defining the 
political issue in a clear and direct manner, two variables were 
employed. The variable was produced by the coder's answer to 
the following questions: 

. ..... 

"Does the respondent state/^the issue specifically 

is the U.S. role in Common Market Consultations" 

(for examination #1) 

"Does the respondent .s;tate that the issue is spe- 
cifically the changing of tenure policy" 

(for examination #2) 
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The second variable was the number of words used in answering 
question one of the test. Since our criterion respondents an- 
swered this question with fewer words than students on the pre- 
test, we hypothesized that there would be fewer words in the 
post-test than in the pre-test. 

The second and third components of political strategy anal- 
ysis skills were the ability to identify political factors affect- 
ing the issue and to suggest political strategies taking into ac- 
count the interests and capabilities of a number of political ac- 
tors. These components were measured by what we term "the weighted 
actor score" of the answer to question 2. This measure involved 
a number of operations. We have reprinted below a description 
of the section of the coding scheme dealing with "weighted ac- 
tors." As the code sheet indicates, there are one of two ways 
to code each political actor mentioned by the respondent. A 
"1" is coded if an actor is mentioned generally in the^answer; - 
a "2" is entered to indicate cases in which the actor is iden-^n - 
tified in the context of a specific strategy suggestion. This 
variable looks at both the frequency and the specificity with 
which the actors are mentioned. It was assumed that the score 
based on thi£> operation would differentiate between those stu- 
dents capable of thinking about political issues in strategic 
terms and those not capable of such analysis. A weighted actor 
score therefore was dependent upon both the number of different 
actors identified in the answer and the degree to which specif ic - * 
strategies were related to those actors. 

Finally, we created a variable that recorded the weighted 
actors excluding the most obvious actor in the two tests, France 
and faculty. This variable is weighted in the same manner as 
the weighted actors but, when arriving at the total number for 
one respondent the main or obvious actor is excluded. The ra- 
tionale behind this procedure is to observe whether students 
cnoose the obvious actor, or whether the increase in political 
sophistication will lead them to choose a wider range of actors. 
The main intent is to measure whether the PRINCE materials will 
broaden the students* conception of the range of the possible 
actors to be dealt with. 

Table 1 provides a summary of the variables we examined in- 
terms of the three-component general political strategy skills 
that we have identified. The hypothesized change in each vari- 
able is also indicated. 
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Table 1: Type of Political Strategy Skill and Indicator 

Variables Used- 



TYPES OF SKILL 

Define Political 
Issues 



VARIABLE DESCRIPTION 
Answer Question #1 

Number of Words Used 
to Answer Question #1 



EXPECTED DIRECTIOI 
OP VARIABLE 

Increase in 
Correct Answers 

Decrease in 
Words 



Identify Political 
Factors 

and 

Articulate Complex 
Political Strategies 



Number of Weighted 
Actors 



Number of Weighted 
Actor Excluding Main 
Actor 



Increase in 
Score 



Increase in 
Score 



Research Design 

The collection of data was dependent upon the cooperation 
of instructors at other colleges who were using the materials 
in regular classes. Furthermore, both time and financial con- 
straints did not permit a selection of test sites on a random 
basis. Hence, in our data collection we worked with a small 
group of instructors who were sufficiently concerned about eval- 
uating the quality of the materials they were using to volunteer 
their cooperation. Our hope originally was to conduct the test 
on ten different campuses allowing us to sample a wide variety 
of post-secondary institutions and to evaluate the use of the 
four materials in various combinations. Although we originally 
had received acceptances by ten instructors, data from only six 
sites reached us in time for this report. Table 2 indicates the 
six testing sites, the number of students and the form of the ma- 
terial used, at each of the institutions. One hundred and fifty- ^ 
one students were included in the sample. We would have preferred 
a larger and more scientifically selected sample of sites. Never- 
theless, we believe that the range of institutional settings and the 
type of data collected contains adequate basis for conducting 
the evaluation we planned. 
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Table 2: Information on the Sample 



School 

Emory University 
Lenior Rhyne College 
Transylvania University 

University of Illinois 

Washington and Lee 
University 

Western Washington 
State College 



Number of 
Students 

32 

47 

21 

11 

12 



28 



Material Used 

PRINCE 

PROBE 

Everyman's PRINCE 
and PRINCEDOWN 

Everyman's PRINCE 
Everyman's PRINCE 
Everyman's PRINCE v 



An experimental single grouped pre-test/post-test design 
was employed . ■'■^ - Instructors were provided only the .instructions, 
contained in Appendix B of this paper. Students were given the 
pre-test prior to the distribution or use of any of the PRINCE 
materials. The post-test was given immediately after the class 
completed the activities associated with the PRINCE materials. 
Thirty-minute response tinie was provided for both the pre. and 
post-tests. At no time during the evaluation were the instruct- 
tors informed how the tests would be coded and analyzed. 



FINDINGS 

One purpose of our evaluation was to see if there were any ^ 
differences between the four types of materials. Our small sam- 
pling of campuses prevented us from making a definitive judgment 
in this respect. However, we did discover variations in the find- 
ings when we dis'aggregated the data and looked at the results 
from each of the six campuses. Figures 3 through 10 display dif- 
ferences between the pre- and post-test for each of the six cam- 
puses on the four variables for the two different tests. The 
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D. T. Campbell, J. C. Stanley, Experimental and Quasi- 
Experimental Designs for Research . (Chicago: Rand McNally & 
Company), p. ?• 
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length of the arrow indicatos the magnitude in the change of the 
means between pre- and post -tests. As can be seen, the general 
pattern is for the post-rtest scores to be closer to the two sets 
of criterion respondents (indicated by the horizontal lines) . 
However, there are some sharp differences between campuses. 

Table 3 presents a different dimension of our findings. In 
that table, each campus site is ranked in descending order of 
"success," as determined by the change in the mean across all 
variables. The ranking takes into account both the direction 
and magnitude of the changes between the pre- and post-tests. 
The table also indicates the materials used, and the amount of 
student hours devoted to the materials. Two patterns are sug- 
gested by the table. The first is that the non-computer version 
of the PRINCE materials, Everyman's PRINCE , was used at the sites 
that were most successful. The other pattern is that the anount 
of time devoted to the materials appears to be positively corre- 
lated with the impact of the materials. Even though there were 
changes in the correct direction across most of the variables 
for all of the testing sites, the differences in amount of time 
spent between the top three and the bottom three are substantial. 
Those differences appear to be related to both the form of ma- 
terials used and the amount of time devoted to the material. It 
should also be noted that at the most successful site, Transylvania 
University, two forms of the material were employed, and the most 
amount of time was devoted to the materials. 

In general, we feel the evidence indicates that the stip- 
ulated educational objectives were being achieved at the campuses 
tested. Table 4 shows the results of combining all 151 respon- 
dents. As that table shows, students responded to an open-ended 
question after being exposed to PRINCE materials with the follow- 
ing differences: 

1) They tended to be more likely to define a clear-cut 
political issue. 

2) They tended to write fewer words in defining the 
issue. 

3) They tended to make more specific suggestions as to 
how to deal with more actors in trying to solve a 
political problem. 
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cod uw wc CD'> ctP'-t 

(Dm Vid). (uro oj'H m-HU 

^SB (DO 505 0)0) 5S>>; 

OS D^tr 06 S.^ S^o) 

TENURE REFORM 

Pre--test Score 46.5 50. 85.6 2.9 1.2 

Post--test Score 39.3 69.7 92.3 3.5 1.6 

Significance* .02 .01 .30 .04 .07 



COMMON MARKET 

Pre-test Score 55.8 73.3 2.9 2.9 1.7 

Post--test Score 45.5 82.7 3.4 3,4 2.0 

Significance* .01 .01 .04 .04 .07 



One-tailed tests of significance^ except for question 
2 mean word count. (See footnote 16.) 
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4) They tended to be more likely to mention actors 
other than the most obvious one in trying to 
solve a political problem. 

In each case, the post-treatment students responded in a way 
as to indicate improved political skills, defined as providing 
answers which were more similar to the responses of the criterion 
respondent in Table 5. For three of the four variables the differ- 
ences were statistically significant, at the .05 level of prob- 
ability of error, according to a one-tailed test. Inspection of 
Table 4 will also show that we have provided the results of the 
word count on question two, the strategy question for the two 
tests. This was not taken to be an indicator of any substantive 
variable. It was recorded as a check to see if there were any 
systematic biases of students* tending to write more or less on 
either test. For both testing instruments the small differences 

between pre-test and post-test were not statistically signifi- 
cant. ^5 



CONCLUSIONS 

At this point in our research, we are left with feelings 
of both encouragement and curiosity. On the one hand, it does 
appear to be possible to develop empirical indicators for some- 
thing reasonably called "political skills" and to teach these 
skills to students. On the other hand, there are wide variations 
in the extent to which the students in this sample achieved an 
improvement on those skills. Several questions occur to us: 

To what extent does the type of student affect success in 
using the materials? (We gathered no data at all on this ques- 
tion.) 

To what extent does student improvement vary with the 
amount of time spent on a given set of materials? (At the most 
successful campus the most time was spent on the materials.) 

To what extent does a combination of two or more of the ma- 
terials reinforce the improvement of the political skills? (At 



The probability of error reported for the number of words 
in the second question is calculated according to a two-tailed 
test, since we are not predicting the direction of the rival hy- 
pothesis. 
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TENURE REFORM 



Question 1 
mean word count 

Question 1 
correct answers 

Question 2 
mean word count 

Mean number of 
weighted actors 

Mean number of weighted 
actors without faculty 



COMMON MARKET 

Question 1 
mean word count 

Question 1 
correct answers 

Question 2 
Bs-jan word count 

Mean number of 
weighted actors 

Me«iii number of weighted 
actors without France 



Western Washington 
State 

31.6 

72.7% 

54.8 

4.5 

3.3 

37.7 

90.0% 
49.8 

3.2 

2.6 



Politicians 
48.1 
60.0% 
131.6 
4.2 
3.1 

35.3 

77.8% 

93.4 

3.6 

2.0 
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the most successful campus there was likewise multijjle use of 
the materials.) 

To what extent does faculty familiarity with, and enthusi- 
asm for, the materials affect their impact? (It has generally 
been found that faculty attitudes about materials are important 
determinants in their educational impact. The faculty members 
in this sample were using the materials voluncarily and were 
therefore presumably positively inclined. We did not measure 
this systematically, however. Furthermore, in all but one case 
the faculty memb.ers used the materials without any training or 
instructions from the authors. The one exception, at the school 
which showed the most change, was a former graduate student of 
the authors.) 

We are hopeful of being able to test some of these questions in 
the future research. 

We undertook an evaluation of PRINCE materials to determine 
if the major pedagogical assumption that we have made in design- 
ing and using th» material is plausible. As mentioned earlier, 
we see the PRINCE materials as an example of social technology 
which can help people understand and cope with the social and 
organizational context of their lives. In constructing and con- 
ducting the evaluation we found that the kind of strategic anal- 
ysis undertaken by experienced political actors is more likely 
to be advocated by inexperienced individuals after they have 
used PRINCE materials. There are still a number of questions 
to be answered concerning the relative merits of the types of 
materials. Most important is the question of whether a multiple 
articulation of a set of ideas has a greater impact than a single 
format. Nevertheless, the basic hypotheses about the impact of 
the materials that we set out to test are, we believe, supported 
by the evidence. 
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APPENDIX A 



INSTRUCTIONS TO THE CODER FOR PRINCE EVALUATION 

I. In the blank space next to the test location number, 
the coder should write the two digit number which ap- 
pears in the top left corner of the questionnaire. 
If a number does not appear, stop and see Larry Bloom 
before continuing. (e.g., 01, 02, etc.) 

II. In the space next to the respondent identification 

number, list the three digit number which appears on 
the top right hand corner of the questionnaire. If 
a number does not appear, stop and see Larry Bloom 
before continuing. (e.g., 001, 002, etc.) 

III. In the blank space next to the year (at college), re- 
cord the appropriate code number from the list below: 

00-fresh. Ol-soph. 02-junior 03-senior 04-all grad. 

students 

OS^Mid-career student informal 06-Mid career student 

formal 

07«Academic criterion respondent 08-Politician criterion 

respondent 09-Business criterion respondent 99-missing 

data 

IV. In the space next to the word sex, place a 0 for Male 
and a 1 for Female. (e.g., sex ) 

V. In the space provided for G.p.A. write the Grade Point 

Average rounded off to the nearest tenth. (e.g., 3.56=36) 
It is important that you omit the decimal point. 

VI. In the space provided for S.A.T. write the Scholastic 
Achievement Test score. It is important that you omit 
the last number. (e.g., 350=35) 

VII. In the space provided for Major, record the appropriate 
code number from the list below: 

1-Political Science 
2«0ther Social Science 
3-Other 
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VIII. In the space provided for coder number record the number 
which has >^een assigned to you* (e.g., coder # 01 ) 

IX. Please make sure that the questionnaire corresponds to 
the coding sheet that you are using. 

X. Questions 1 and 2 are simply the number of words in che 
respondent's answer. Count the total number of words 
and record this number in *:he space provided. 

XI. If the respondent states the issue specifically, place 
a 0 in the space provided. If the respondent does not 
state the issue specifically, place a 1 in the space 
provided. (e.g.. Common Mo.rket-1*he U.S. role in Common 
Market Consultation. = 0 ) (e.g.. Common Market-The 
Respondent does not in any way st3te the U.S. role in 
Common Market Consultation = 1 ) (^^.g*. Tenure Reform- 
The Respondent specifically states the change, revision 
or altering of Tenure policy = 0 ) (e.g.. The Respon- 
dent does not in any way specifically state the change, 
revision or altering of Tenure Policy = 1 ) 

XII. In the space provided for each actor, record a 0 if the 
actor is not mentioned. 

If the actor is mentioned in the answer to question #2, 
place a 1 next to the actor's name on the coding sheet* 
(If the actor's name does not appear on the coder sheet, 
place a 1 in the space provided next to * Other' and write- 
the actor's name next to the column number.*) 

If the actor is mentioned and a general statement follows, 
the coder should also place a 1 in the space provided. 
(Refer to the * above for Actors not mentioned on coding 
sheet.) (e.g.. Secretary of State Kissinger should talk 
with France.) 

If the actor is mentioned and a specific statement follows, 
place a 2 next to the Actor's name on coding sheet. (Re- 
fer to the * above for Actor's name on coding sheet.) 
(e.g.. Secretary of State Kissinger bargains with France 
to attempt to coerce the French into accepting the U.S. 
position on the Middle East dilemma-) 

XIII. If there are any problems and/or questions, see Larry 
Bloom immediately. 
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PRINCE EVALUATION CODING SHEET 
FOR TENURE REFORM ARTICLE 



Test Location (col. 1,2) Respondent # (^col.3,4,5) 

Year _(col.6,7) Sex (col. 8) G.P.A. (col. 9, 10) 

S.A.T._ (col..ll,12) Major. (col. 13) Coder # (col. 14, 15) 

Questionnaire # 0 (col. 16) 

1. Number of words in question #1 (col . 17, 18, 19) 

2. Number of words in question #2 (col . 20, 21, 22) 

3. Does the respondent state that the issue is specifically 
the changing of the tenure policy? (col. 23) 

4. Actors 

1. Board of Trustees ^ (col. 24) 

2. SUNY faculty generally (col. 25) 

3. Tenured SUNY faculty (col. 26) 

4. Untenured SUNY faculty (col. 27) 

5. Faculty in high enrollment courses (col. 28) 

6. Faculty in low enrollment courses (col. 29) 

7. High prestige faculty (col. 30) 

8. SUNY Administrators (col. 31) 

9. Students (col. 32) 

10. Teachers Unions (col. 33) 

LL. News Media (col. 34) 

12. General Public (col. 35) 

13. Legislators (col. 36) 

14. Ad Hoc Committee (col. 37) 

15. Commission (col. 38) 

16. Research Group (col. 39) 

17. Other (col. 40) 

18. Other (col. 41) 

19. Other (col. 42) 

20. Other (col. 43) 

21. Other (col. 44) 

22. E veryman's PRINCE (col. 45) 

23. PRINCE- (<iol:46) ) 

24. PROBE (col. 47) 

25. Oral PRINCE (col. 48) 

26. Test (col. 49) 
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PRINCE EVALUATION CODING SHEET 
FOR COMMON MARKET ARTICLE 



Test Location (col.1,2) Respondent # (col. 3, 4, 5) 

Year (col. 6, 7) Sex (col. 8) G.P.A. ^(col.9,10) 

S.A.T. (col. 11, 12) Major (col. 13) Coder # c (col. 14, 15] 

Questionnaire # (col. 16) 



1. Number of words in question #1 _(col. 17, 18,19) 

2. Number of words in question #2 (col. 20, 21,22) 

3. Does the respondent state the issue specifically in the 
U.S. role in Common Market Consultations? (col. 23) 



Actors 






1. 


France 


(col. 24) 


2. 


West Germany (col. 25) 


3. 


Britain 


(col. 26) 


4. 


All other members of the Common Market 


5. 


Other 


(col. 28) 


6. 


Other 


(col. 29) 


7. 


Other 


(col. 30) 


8. 


Other 


(col. 31) ■ 


9. 


Other 


(col. 32) 


10. 


Other 


(col. 33) 


11. 


Columns 


34-44 punch zero 


12. 


Everyman 


's PRINCE (col. 45) 


13. 


PRINCE 


(col. 46) 


14. 


PROBE 


(col. 47) 


15. 


Oral PRINCE (col. 48) 


16. 


Test 


(col. 49) 
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APPENDIX B 



INSTRUCTIONS FOR ADMINISTERING PRE- AND POST-TESTS FOR EVALUAT- 
ING STUDENT RESPONSES TO A PROBLEM IN POLITICAL STRATEGY 

1. When administering the pre-test^ divide the group in half 
according to some convenient randomizing procedure — seating 
location (if it is assigned) , by alphabet according to stu- 
dent's last name^ or some other procedure. However, please 
note no, *3 below ! Students who take one test prior to the 
Exposure to PRINCE materials must take the other test sub- 



sequent to their exposure. 

2. The time allocated for the test should not exceed 30 
minutes, beginning when the students receive the exam. 

3. Ask each student to complete the personal information, as 
well as answering the two questions concerning the New York 
Times article. 

4. After the PRINCE exercise or exercises have been completed, 
administer the post- test as soon as possible. Be sure that 
students who took the tenure reform article as the pre-test 
are given the Common Market article as the post-test, and 
vice-versa. Students should fill out the personal informa- 
tion on the post-test. 

5. As soon as either set of tests is completed, please return 
them to: 



6. please enclose with the post- tests a brief stateinent of the 
nature and extent of the PRINCE-related exercises which the 
students undertook. • The attached form can be checked as 
appropriate to provide this information. 




Michael K. O'Leary 
International Relations Program 
712 Ostrom Avenue 
Syracuse University 
Syracuse, New York 13210 
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